De-multiplexing streaming data from a fast write ahead durable log into segments

ABSTRACT

Implementations are provided herein for managing streaming data that appended and multiplexed into a durable write ahead replicated log. By writing data in the durable log, large amounts of small writes can be processed quickly. Data in the durable log can be de-multiplexed and packaged into segment containers. Segment containers can serialize stream segment specific data and can be stored in long term storage. Data that has been stored in long term storage can be truncated from the durable write ahead log making room for new data.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.62/503,845 filed May 9, 2017.

FIELD OF THE INVENTION

This invention relates generally to processing data, and moreparticularly to processing streaming data.

BACKGROUND OF THE INVENTION

Streaming applications are applications that deal with a large amount ofdata arriving continuously. In processing streaming application data,the data can arrive late, arrive out of order, and the processing canundergo failure conditions. It can be appreciated that tools designedfor previous generations of big data applications may not be ideallysuited to process and store streaming application data.

Enabling streaming applications to store large amounts of data from astorage perspective can be challenging. There is a need to determine theproper storage primitive that would ideally be suited for building a newgeneration of streaming applications in conjunction with existing toolslike Apache Flink. In using a Lambda architecture, a developer may use acomplex combination of middleware tools that include batch stylemiddleware influenced by platforms like Apache Hadoop and continuousprocessing tools like Apache Storm, Apache Samza, Apache Kafka andothers. Batch style processing can be used to deliver accurate butpotentially out of data analysis of data. It can be appreciated that“real-time” processing may deliver faster results but could come at acost of reduced accuracy. Furthermore, there may be a need for twocopies of application logic because the programming models of a speedlayer are different than those used in a batch layer.

Conventionally, Lambda architectures may be expensive to develop andexpensive to deploy and manage in production. In some implementations,as more application like Internet of Things (“IoT”), require continuousprocessing, it may not be beneficial to use Lambda architectures andconventional style middleware. Therefore, there exists a need for asimpler approach to Lambda to process streaming application data.

SUMMARY

The following presents a simplified summary of the specification inorder to provide a basic understanding of some aspects of thespecification. This summary is not an extensive overview of thespecification. It is intended to neither identify key or criticalelements of the specification nor delineate the scope of any particularimplementations of the specification, or any scope of the claims. Itssole purpose is to present some concepts of the specification in asimplified form as a prelude to the more detailed description that ispresented in this disclosure.

In accordance with an aspect, a set of segment containers can bemaintained, wherein each segment container in the set of segmentcontainers is associated with at least one segment, and each segment isassociated with one segment container. A set of operations received froma client application can be assigned to segment containers within theset of segment containers using a predefined mapping function. A set ofoperations can be read from a durable log in a segment container,wherein the durable log contains multiplexed data associated with a setof stream segments that are mapped to the segment container. The set ofoperations can be de-multiplexed into a set of stream segment buffersbased on a stream segment associated with each operation in the set ofoperations in the durable log. Data within a segment among the set ofsegments, within a segment container among the set of segmentcontainers, can be packaged, based on the set of stream segment buffers,in response to a policy trigger. The packaged data can be stored in along term storage location.

The following description and the drawings set forth certainillustrative aspects of the specification. These aspects are indicative,however, of but a few of the various ways in which the principles of thespecification may be employed. Other advantages and novel features ofthe specification will become apparent from the detailed description ofthe specification when considered in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example illustration of a Lambda architecture forprocessing streaming application data;

FIG. 2 illustrates an example illustration of a Pravega architecture forprocessing streaming application data in accordance with implementationsof this disclosure;

FIG. 3 illustrates an example of multiple individual applications orjobs in a streaming application data pipeline in accordance withimplementations of this disclosure;

FIG. 4 illustrates an example data processing environment for processingstreaming application data in a conventional system;

FIG. 5 illustrates an example data processing environment for processingstreaming application data in accordance with implementations of thisdisclosure;

FIG. 6 illustrates an example Pravega Cluster in accordance withimplementations of this disclosure;

FIG. 7 illustrates an example Pravega Cluster ingesting data from a setof writers and making data available to a set of reader groups inaccordance with implementations of this disclosure;

FIG. 8 illustrates an example Stream decomposed into a set of StreamSegments in accordance with implementations of this disclosure;

FIG. 9 illustrates an example graph depicting the relationship betweenrouting keys, stream segments, and time in accordance withimplementations of this disclosure;

FIG. 10 illustrates an example graph depicting Events and theirrelationship between routing keys, stream segments, and time inaccordance with implementations of this disclosure;

FIG. 11 illustrates an example of Stream Segments and associatedReaderGroups in accordance with implementations of this disclosure;

FIG. 12 illustrates an example graph depicting Events published into aTransaction and their relationship between routing keys, streamsegments, and time in accordance with implementations of thisdisclosure;

FIG. 13 illustrates an example state synchronizer in accordance withimplementations of this disclosure;

FIG. 14 illustrates an example implementation of a set of streamingclients, a Pravega Cluster, the autoscaling of segments, and tieredstorage in accordance with implementations of this disclosure;

FIG. 15 illustrates an example append-only log in accordance withimplementations of this disclosure; and

FIG. 16 illustrates an example flow diagram method for de-multiplexingstreaming data from a durable log into a set of segment containers andstoring data form the segment containers into long term storage inaccordance with implementations of this disclosure.

DETAILED DESCRIPTION

The innovation is now described with reference to the drawings, whereinlike reference numerals are used to refer to like elements throughout.In the following description, for purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of this innovation. It may be evident, however, that theinnovation can be practiced without these specific details. In otherinstances, well-known structures and devices are shown in block diagramform in order to facilitate describing the innovation.

As used herein, the term “node” refers to a physical computing device,including, but not limited to, network devices, servers, processors,cloud architectures, or the like. In at least one of the variousimplementations, nodes may be arranged in a cluster interconnected by ahigh-bandwidth, low latency network backplane. In at least one of thevarious implementations, non-resident clients may communicate to thenodes in a cluster through high-latency, relatively low-bandwidth frontside network connections, such as Ethernet, or the like.

As used herein, the terms “Stream” or “Pravega” refers to a new storageprimitive. A stream is ideally suited to the continuous processing ofunbounded data. In many implementations, a stream may be a name,durable, append-only and unbounded sequence of bytes. In mostimplementations, with this primitive, and with the key featuresdiscussed in this disclosure, the stream can be combined with streamprocessing engines such as Apache Flink to build streaming applications.

Implementations are provided herein for managing streaming data that isappended and multiplexed into a durable write ahead replicated log. Bywriting data in the durable log, large amounts of small writes can beprocessed quickly. Data in the durable log can be de-multiplexed andpackaged into segments. Segments can serialize stream segment specificdata and can be stored in long term storage. Data that has been storedin long term storage can be truncated from the durable write ahead logmaking room for new data.

Referring now to FIG. 1, there is illustrated an example illustration ofa Lambda architecture for processing streaming application data. In someprior implementations, in using a Lambda architecture, a developer mayuse a complex combination of middleware tools that include batch stylemiddleware influenced by platforms like Apache Hadoop and continuousprocessing tools like Apache Storm, Apache Samza, Apache Kafka andothers. Batch style processing can be used to deliver accurate butpotentially out of data analysis of data. It can be appreciated that“real-time” processing may deliver faster results but could come at acost of reduced accuracy. Furthermore, there may be a need for twocopies of application logic because the programming models of a speedlayer are different than those used in a batch layer. With duplicatedstorage and middleware in both the speed layer and the batch layer, itcan be difficult to determine the source that has the actual source ofcorrect real time data. Merging the logic of the speed layer and thebatch layer in the serving layer that interfaces with the applicationclient can also be challenging. Thus, conventionally, Lambdaarchitectures may be expensive to develop and expensive to deploy andmanage in production.

FIG. 2 illustrates an example illustration of a Pravega architecture forprocessing streaming application data in accordance with implementationsof this disclosure. In many implementations, there can be one set ofmiddleware to deploy and operate. In some implementations, there can beone expression of application logic to develop and maintain rather thantwo. There can be one ingest process from the source data into Pravega.One source of truth for data can lie within Pravega. There can be oneembodiment of application. There can be one set of middleware. It can beappreciated that this can create a more stable and a simpler to developfor application environment for the Application Client.

FIG. 3 illustrates an example of multiple individual applications orjobs in a streaming application data pipeline in accordance withimplementations of this disclosure. It can be appreciated that in mostimplementations, most applications, including Internet of Things (“IoT”)applications, involve a pipeline of several, often many more individualapplications or jobs. IN some implementations, a Kappa environment mayhave requirements imposed on middleware in terms of chains of pipelinesof data processing. For example, some requirements may involve end-endsemantics in addition to component by component capabilities. In someimplementations, the current disclosure considers scaling in terms ofend-end scaling, in addition to scaling a single component. In certainimplementations, the current disclosure considers ordering, exactly oncedelivery semantics, fault tolerance, etc., across the entire pipeline.In some implementations, the current disclosure can enable coordinationof the output of one component with the input of a downstream componentto deliver a pipeline of stream components more efficiently.

Exactly Once Semantics

In some implementations, exactly once semantics can mean the currentdisclosure can ensure that data is delivered and processed exactly once,with exact ordering guarantees, despite failures in clients, servers, orthe network. In certain implementations, to achieve exactly oncesemantics, Pravega Streams may be durable, ordered, consistent andtransactional.

In most implementations, ordering can mean that data is seen by readersin the order it is written. In some implementations of the currentdisclosure, data may be written along with an application-definedrouting key. In certain implementations, the current disclosure may makeordering guarantees in terms of routing keys. In many implementations,two pieces of data with the same routing key may be read by a reader inthe order they were written. In most implementations, the currentdisclosures may enable ordering guarantees allow data reads to bereplayed (e.g. when applications crash) and the results of replaying thereads may be the same.

In most implementations, consistency may mean that Readers see the sameordered view of data for a given routing key, even in the face offailure. In most implementations, systems that are “mostly consistent”may not sufficient for building accurate data processing.

Conventionally, systems that provide “at least once” semantics maysuffer from the possibility of data duplication. In typical systems, incertain failure scenarios, a data producer, upon recovery may write thesame data twice. In most implementations of the current disclosure,writes may be idempotent, rewrites done as a result of failure recoverymay not result in data duplication. In most implementations, it may notbe possible to deliver exactly once processing semantics when there isthe possibility of data duplication.

In certain implementations, techniques may extend beyond just exactlyonce semantics between two components within an applicationarchitecture, and may include end-end exactly once semantics across anentire pipeline of streaming application components with strict orderingof data in the face of component failures. In some implementations, datastorage may be the foundation for once across an entire data processingpipeline. In many implementations, data storage may be important forbuilding streaming applications that are both timely and accurate. Inmost implementations, without accuracy at the storage layer, applicationdevelopers may be faced with relying on the complexity of a Lambdaarchitecture to deliver accurate data analysis in combination with nearreal-time processing of the data.

Auto Scaling

Unlike systems with static partitioning, the current disclosure mayautomatically scale individual data streams to accommodate changes indata ingestion rate.

In a particular implementation, imagine an IoT application with millionsof devices feeding thousands of data streams with information aboutthose devices. In this implementation, imagine a pipeline of Flink jobsthat process those streams to derive business value from all that rawIoT data: predicting device failures, optimizing service deliverythrough those devices, or tailoring a customer's experience wheninteracting with those devices. In this implementation, building such anapplication at scale is difficult without having the components be ableto scale automatically as the rate of data increases and decreases.

Conversely, with implementations of the current disclosure, it may beeasy to elastically and independently scale data ingestion, storage andprocessing—orchestrating the scaling of every component in a datapipeline. In many implementations, the current disclosure may supportauto-scaling starts with the idea that Streams are partitioned intoStream Segments. In most implementations, a Stream may have 1 or moreStream Segments, any data written into the Stream may be written to oneof the Stream Segments based on a routing key. In certainimplementations, writers may use application-meaningful routing keyslike customer-id, timestamp, machine-id, so like data is groupedtogether.

In some implementations, a Stream Segment may be a fundamental unit ofparallelism in Pravega Streams. In many implementations, a Stream withmultiple Stream Segments may support more parallelism of data writes;multiple Writers writing data into the different Stream Segmentspotentially involving all the Pravega Servers in the cluster. In certainimplementations, on the Reader side, the number of Stream Segments mayrepresent the maximum degree of read parallelism possible. In certainimplementations, if a Stream has N Stream Segments, then a ReaderGroupwith N Readers may consume from the Stream in parallel. In mostimplementations, increasing the number of Stream Segments may increase anumber of Readers in a ReaderGroup to increase the scale of processingthe data from that Stream. In further implementations, as the number ofStream Segments decreases, the number of Readers may be reduced.

In some implementations, a Stream may be configured to grow the numberof Stream Segments as more data is written to the Stream, and to shrinkwhen data volume drops off. In certain implementations, growing andshrinking a stream may be referred to herein as the Stream's ServiceLevel Objective or SLO. In many implementations, the current disclosuremay enable monitoring a rate of data input to the Stream and uses theSLO to add or remove Stream Segments from a Stream. In certainimplementations, segments may be added by splitting a Segment. Infurther implementations, segments may be removed by merging twoSegments. In most implementations, the number of Stream Segments mayvary over time.

In certain implementations, it may be possible to coordinate the autoscaling of Streams in Pravega with application scale out. In furtherimplementations, using metadata available from Pravega, applications mayconfigure the scaling of their application components. In a particularimplementation, scaling may drive the number of instances of a Flinkjob. In another implementation, it may be possible to use software suchas Cloud Foundry, Mesos/Marathon, Kubernetes or the Docker stack todeploy new instances of an application to react to increased parallelismat the Pravega level, or to terminate instances as Pravega scales downin response to reduced rate of data ingestion.

Distributed Computing Primitive

In most implementations, the current disclosure may enable efficient useof distributed applications, such as micro-services. In certainimplementations, the current disclosure may enable data as a datastorage mechanism, for messaging between micro-services and for otherdistributed computing services such as leader election.

In some implementations, a State Synchronizer may be a basis of sharingstate across a cluster with consistency and optimistic concurrency. Infurther implementations, a State Synchronizer may be based on afundamental conditional write operation, so that data may be written ifit would appear at a given position in the Stream. In certainimplementations, if a conditional write operation cannot meet thecondition, it may fail. In certain implementations, a State Synchronizermay be a strong synchronization primitive that may be used for sharedstate in a cluster, membership management, leader election and otherdistributed computing scenarios.

Write Efficiency

In many implementations, the current disclosure may enable shrinking ofwrite latency to milliseconds, and may seamlessly scale to handle highthroughput reads and writes from thousands of concurrent clients, makingit ideal for IoT and other time sensitive applications.

In implementations of the current disclosure, streams may be lightweight, and implementations of the current disclosure may be able tosupport millions of Streams. In many implementations, the techniques ofthe current disclosure may free an application from worrying aboutstatically configuring streams and pre allocating a small fixed numberof streams and husbanding or limiting stream resource.

In many implementations, the current disclosure may enable low latencyWrite operations, for example, under 10 ms to return an acknowledgementto a Writer. In further implementations, writes may be optimized so thatI/O throughput may be limited by network bandwidth rather than apersistence mechanism being the bottle neck. In certain implementations,Apache BookKeeper may be used to persist all write operations. In someimplementations, BookKeeper may persist and protect the data veryefficiently. In certain implementations, as data may be protected beforethe write operation is acknowledged to the Writer, data may be durable.In many implementations, data durability may be a fundamentalcharacteristic of a storage primitive. In many implementations, to addfurther efficiency, writes to BookKeeper may often involve data frommultiple Stream Segments, so the cost of persisting data to disk may beamortized over several write operations. In most implementations of thecurrent disclosure, there may be no durability-performance trade-off aswrites may be both durable and high performance.

In most implementations, reads may be efficient. In certainimplementations, a reader may read from a Stream either at the tail ofthe Stream or at any part of the Stream's history. Unlike some log-basedsystems that use the same kind of storage for tail reads and writes aswell as reads to historical data, implementations of the currentdisclosure may use two types of storage. In many implementations, a tailof a Stream may be in so-called Tier-1 storage. In most implementations,writes may be implemented by Apache BookKeeper as described herein. Incertain implementations, tail reads may be served out of a managedmemory cache. In fact, in some implementations BookKeeper may servereads in failure recovery scenarios, for example, when a crash hasoccurred and the system is being recovered. In some implementations, thehistorical part of the Stream may be in Tier-2 storage that may beoptimized for low cost storage with high throughput. In someimplementations, Pravega may use efficient in-memory read ahead cache,taking advantage of the fact that Streams are usually read in largecontiguous chunks and that HDFS is well suited for those sort of large,high throughput reads. In most implementations, tail reads may notimpact the performance of writes.

Infinite Retention

In many implementations, data in Streams may be retained indefinitely.In most implementations, the current disclosure enables an API to accessboth real-time and historical data. In certain implementations, batchand real-time applications may be handled efficiently.

In some implementations, there may be a value to keeping old data. Insome implementations, for example in machine learning, it may bedesirable to periodically change the model and train the new version ofthe model against as much historical data as possible to yield a moreaccurate predictive power of the model. In many implementations, thecurrent disclosure has auto-tiering and enables keeping historical datathat does not affect the performance of tail reads and writes.

In many implementations, the size of a stream may not be limited by thestorage capacity of a single server, but rather, may be limited by thestorage capacity of a data center or a cloud provider. In furtherimplementations, as cost of storage decreases, the economic incentive todelete data may go away.

Storage Efficiency

In many implementations, the current disclosure enables buildingpipelines of data processing, combining batch, real-time and otherapplications without duplicating data for every step of the pipeline.

FIG. 4 illustrates an example data processing environment for processingstreaming application data in a conventional system. Consider a dataprocessing environment as depicted that combines real time processingusing Spark, Flink, and or Storm; Hadoop Distributed File System(“HDFS”) for batch processing; some kind of Lucene-based Searchmechanism like Elastic Search for full text search; and maybe one (orseveral) NoSQL databases to support micro-services apps.

Using traditional approaches, one set of source data, for example sensordata from an IoT app, would be ingested and replicated separately byeach system. Conventionally there would be with 3 replica copies of thedata protected in the pub/sub middleware system, 3 copies of the data inHDFS, 3 copies of the data in Lucene, 3 copies of the data in the NoSQLdatabase, and potential 3 copies of the data in other support servicesdesirable to the system. When we consider the source data is measured interabytes, the cost of data replication separated by middleware categorybecomes prohibitively expensive.

FIG. 5 illustrates an example data processing environment for processingstreaming application data in accordance with implementations of thisdisclosure. In this implementation, consider the same pipeline from FIG.4 using techniques of the current disclosure and middleware adapted touse techniques of the current disclosure for its storage.

In implementations of the current disclosure, there may be a singlesource of truth for data processing. In many implementations, with tier2 storage, and features like erasure coding, 1 TB of data may beprotected using less than 2 TB of storage. In further implementations,data may be ingested and protected in one place. Implementations of thecurrent disclosure enable one place for storage and may provide a singlesource of truth for the entire pipeline. In further implementations, ifthe bulk of the data being stored in Tier-2 locations that are enabledwith erasure coding to efficiently protect the data, the storage cost ofthe data may be substantially reduced.

Durability

In some implementations, there may not be compromise betweenperformance, durability and consistency. In many implementations, thecurrent disclosure may enable durable storage of streaming data withstrong consistency, ordering guarantees and great performance. In mostimplementations, durability may be a fundamental storage primitiverequirement. Conventionally, storage that could lose data is notreliable storage. Usually, systems based on storage can lose data maynot be production quality.

In some implementations, once a write operation is acknowledged, data ofthe write may not be lost, even when failures occur. In manyimplementations, data may not be lost as data may be saved in protected,persistent storage before the write operation returns to the Writer. Inmost implementations, data in a Stream may be protected. In manyimplementations, a Stream may be treated as a system of record, similarto databases or files.

Transaction Support

In many implementations, the current disclosure may ensure that a set ofevents are written to a stream atomically.

In some implementations, a Pravega Transaction may be part of aPravega's writer API. In certain implementations, data may be written toa Stream through the API, or an application may write data through aTransaction. In some implementations, with Transactions, a Writer maypersist data at a one point in time, and later decide whether the datashould be appended to a Stream or abandoned.

In many implementations, using a Transaction, data may be written to aStream when the Transaction is committed. In most implementations, whena Transaction is committed, data written to the Transaction may beatomically appended to the Stream. In many implementations, asTransactions may be implemented in the same way as Stream Segments, datawritten to a Transaction is just as durable as data written directly toa Stream. In certain implementations, if a Transaction is abandoned(e.g. if the writer crashes) the Transaction may be aborted and all datamay be discarded. In some implementations, an application may choose toabort the Transaction through the API if a condition occurs thatsuggests the writer should discard the data.

In many implementations, transactions may enable chaining Flink jobstogether. In some implementations, when a Flink job uses Pravega as asink, it may begin a Transaction, and if it successfully finishesprocessing, it may commit the Transaction, writing the data into a basedsink. In some implementations, if the job fails a Transaction may timeout and data may not be written. In some implementations, when a job isrestarted, there may be no “partial result” in a sink that may need tobe managed or cleaned up.

In some implementations, by combining Transactions, it may be possibleto chain Flink jobs together, having one job's Pravega-based sink be thesource for a downstream Flink job. In some implementations, the currentdisclosure may provide the ability for a pipeline of Flink jobs to haveend-end exactly once, guaranteed ordering of data processing.

In some implementations, it may be possible for Transactions acrossmultiple Streams to be coordinated with Transactions, so that a Flinkjob may use 2 or more Pravega-based sinks to provide source input todownstream Flink jobs. In additional implementations, it may be possiblefor application logic to coordinate Pravega Transactions with externaldatabases such as Flink's checkpoint store.

Example Distributed System

FIG. 6 illustrates an example Pravega Cluster in accordance withimplementations of this disclosure. Implementations of the currentdisclosure may be deployed as a distributed system. In a particularimplementation, there may be a cluster of servers and storagecoordinated to run Pravega called a “Pravega cluster.” In someimplementations, implementations of the current disclosure may becomposed of two kinds of workload: Controller instances and PravegaNodes.

In some implementations, a set of Controller instances may make up acontrol plane, which may provide functionality to create, alter anddelete Streams, retrieve information about Streams, monitor the healthof a Pravega cluster, gather metrics etc. In most implementations, theremay be multiple (recommended at least 3) Controller instances running ina cluster for high availability.

In some implementations, a set of Pravega Nodes in a cluster maycomprise a data plane or Pravega data plane. In many implementationsservers, or Pravega Nodes, may provide an API to read and write data inStreams. In certain implementations, data storage may be comprised oftwo tiers: Tier 1 Storage, which provides short term, low-latency, datastorage, guaranteeing the durability of data written to Streams and Tier2 Storage providing longer term storage of Stream data. In certainimplementations, Apache Bookkeeper may be used to implement Tier 1Storage and can use HDFS, DellEMC's Isilon OneFS or DellEMC's ElasticCloud Storage (ECS) to implement Tier 2 Storage. In someimplementations, Tier 1 Storage may run within a Pravega cluster. Inmost implementations, Tier 2 Storage may be deployed outside the Pravegacluster.

In many implementations, tiering storage may be important to deliver thecombination of fast access to Stream data but also allow Streams tostore a vast amount of data. In some implementations, Tier 1 storage maypersist the most recently written Stream data. In some implementations,as data in Tier 1 Storage ages, it may be moved into Tier 2 Storage.

Implementations of the current disclosure may use Apache Zookeeper as acoordination mechanism for the components in the Pravega cluster. Inother implementations, other components, including Pravega componentssuch as BookKeeper may rely on Zookeeper.

In some implementations, the current disclosure may provide a clientlibrary, written in Java, for building client-side applications such asanalytics applications using Flink. In some implementations herein, aclient library may be referred to as a Pravega client library. Incertain implementations, a Pravega Java Client Library may manage theinteraction between application code and Pravega via a custom TCP wireprotocol.

Streams

In certain implementations, the current disclosure may organize datainto Streams. In most implementations, a stream may be a persistent,unbounded, append-only collection of data records called Events. In someimplementations herein, a stream may be referred to as a Pravega Stream.

In certain implementations, a Pravega Stream may be based on anappend-only log data structure. In some implementations, by usingappend-only logs, the current implementation enable rapid ingestion ofinformation into durable storage, and support a large variety ofapplication use cases such as publish/subscribe messaging similar toApache Kafka, NoSQL databases such as a Time Series Database (TSDB),workflow engines, event-oriented applications and many other kinds ofapplications.

In some implementations, when a Stream is created, the Stream may begiven a meaningful name such as “IoTSensorData” or“WebApplicationLog_201 7-03-30T13:08:10.” In certain implementations, aStream's name may help other developers understand the kind of data thatis stored in the Stream. In most implementations, a Pravega Stream namesmay be organized within a Scope. In most implementations, a Scope may bea string and may convey some sort of meaning to developers such as“FactoryMachines” or “HR_Website_logs.” In some implementations, a Scopemay act as a namespace for Stream names. Stream names can be uniquewithin a Scope.

In many implementations, a Stream may be uniquely identified by acombination of its Stream Name and Scope. In certain implementations,Scope may be used to segregate names by tenant (in a multi tenantenvironment), by department of an organization, by geographic locationor any other categorization the developer chooses.

In some implementations, a Stream may be unbounded in size. In manyimplementations, there may not be a limit on how many Events can be in aStream or how many total bytes are stored in a Stream. In mostimplementations, Pravega may be a data storage primitive. In certainimplementations, the amount of data stored in Pravega may be limited bythe total storage capacity of the data center. In most implementations,once data is written to it, it may be durably stored.

In some implementations, to deal with a potentially large amount of datawithin a Stream, Pravega Streams may be divided into Stream Segments. Incertain implementations, a Stream Segment may be a shard, or partitionof the data within a Stream.

Events

In many implementations, a primitive data element in Pravega may be anEvent. In some implementations, an Event may be a collection of byteswithin a Stream. In certain implementations, an Event may be as simpleas a small number of bytes containing a temperature reading from an IoTsensor composed of a timestamp, a metric identifier and a value. In someimplementations, an event may be web data associated with a user clickon a website. In certain implementations, an event may be anything youcan represent as a collection of bytes. In many implementations,applications may make sense of Events using standard Java serializersand deserializers, allowing them to read and write objects in Pravegausing similar techniques to reading and writing objects from file-basedstorage.

In most implementations, an Event may have a Routing Key. In certainimplementations, a Routing Key may be a String that allows Pravega andapplication developers to determine which Events are related. In furtherimplementations, a Routing Key may be a string that developers use togroup similar Events together. In certain implementations, a Routing Keymay be derived from data naturally occurring in an Event, something like“customer-id” or “machine-id”, but it may also be some artificial Stringsuch as a UUID or a monotonically increasing number. In manyimplementations, a Routing Key may be something like a timestamp (togroup Events together by time) or perhaps a Routing Key could be an IoTsensor id (to group Events by machine). In certain implementations, aRouting Key may be important to defining precise read and writesemantics.

Readers, Writers and ReaderGroups

FIG. 7 illustrates an example Pravega Cluster ingesting data from a setof writers and making data available to a set of reader groups inaccordance with implementations of this disclosure. In someimplementations, the current disclosure provides for a client library,written in Java that may implement a convenient API for Writer andReader applications to use. In some implementations, a Pravega JavaClient Library may encapsulate a wire protocol used to communicatebetween Pravega clients and Pravega.

In some implementations, a Writer may be an application that createsEvents and writes them into a Stream. In most implementations, data maybe written by appending to the tail (front) of a Stream.

In certain implementations, a Reader may be an application that readsEvents from a Stream. In many implementations, a Reader may read fromany point in the Stream. In certain implementations, a Reader may bereading Events from the tail of the Stream. In certain implementations,Events may be delivered to Readers as quickly as possible. For example,in one implementation, Events can be delivered to Readers within tens ofmilliseconds after they were written. In further implementations,Readers may read from earlier parts of the Stream (called catch-upreads). In certain implementations, a Position may represent point in aStream a Reader is currently located. In some implementations, aPosition object may be used as a recovery mechanism—applications thatpersist the last Position a Reader has successfully processed can usethat information to initialize a replacement Reader to pickup where afailed Reader left off. In certain implementations, using this patternof persisting Position objects, an application may be built thatguarantees exactly once Event processing in the face of Reader failure.

In some implementations, Readers may be organized into ReaderGroups. Inmany implementations, a ReaderGroup may be a named collection of Readersthat together, in parallel, read Events from a given Stream. In certainimplementations, when a Reader is created through the Pravega data planeAPI, the name of the ReaderGroup may be part of it. In certainimplementations, each Event published to a Stream may be guaranteed tobe sent to one Reader within the ReaderGroup. In many implementations,there may be one Reader in the ReaderGroup or there may be many. Infurther implementations, there may be many different ReaderGroupssimultaneously reading from any given Stream.

In certain implementations, ReaderGroup may be a “composite Reader” or a“distributed Reader”, that allows a distributed application to read andprocess Stream data in parallel, such that a massive amount of Streamdata can be consumed by a coordinated fleet of Readers in a ReaderGroup.In some implementations, a collection of Flink tasks processing Streamdata in parallel may be an example use of a ReaderGroup.

Stream Segments

FIG. 8 illustrates an example Stream decomposed into a set of StreamSegments in accordance with implementations of this disclosure. In manyimplementations, a Stream may be decomposed into a set of StreamSegments. A Stream Segment is a shard or partition of a Stream. In manyimplementations, an Event may be Stored within a Stream Segment

In some implementations, a Stream Segment may be a container for Eventswithin the Stream. In certain implementations, when an Event is writteninto a Stream, it may be stored in one of the Stream Segments based onthe Event's Routing Key. In certain implementations, consistent hashingmay be used to assign Events to Stream Segments. In certainimplementations, event Routing Keys may be hashed to form a “key space.”In some implementations, key space may be divided into a number ofpartitions, corresponding to the number of Stream Segments. In someimplementations, consistent hashing may determine which Segment an Eventmay be assigned to.

Auto Scaling Stream Segments

In some implementations, AutoScaling may refer to a number of StreamSegments that vary over time. In some implementations, a number ofStream Segments in a Stream may grow and shrink over time as I/O load onthe Stream increases and decreases, which may be referred to herein asAutoScaling.

FIG. 9 illustrates an example graph depicting the relationship betweenrouting keys, stream segments, and time in accordance withimplementations of this disclosure. In this example implementation, aStream starts out at time t₀ with a configurable number of Segments. Ifthe rate of data written to the Stream is constant, the number ofSegments won't change. However at time t1, the system noted an increasein the ingestion rate and chose to split Segment 1 into two parts. Wecall this a Scale-up Event. Before t1, Events with a Routing Key thathashes to the upper part of the key space (values 200-399) would beplaced in Segment 1 and those that hash into the lower part of the keyspace (values 0-199) would be placed in Segment 0. After t1, Segment 1is split into Segment 2 and Segment 3. Segment 1 is sealed and it nolonger accepts writes. At this point in time, Events with Routing Key300 and above are written to Segment 3 and those between 200 and 299would be written into Segment 2. Segment 0 still keeps accepting thesame range of Events as before t1.

In this implementation, there is another Scale-up Event at time t₂, asSegment 0's range of Routing Key is split into Segment 5 & Segment 4.Also at this time, Segment 0 is sealed off so that it accepts no furtherwrites. Segments covering a contiguous range of the key space can alsobe merged. At time t3, Segment 2's range and Segment 5's range aremerged into Segment 6 to accommodate a decrease in load on the Stream.

In some implementations, when a Stream is created, it may be configuredwith a Scaling Policy that determines how a Stream reacts to changes inits load.

In certain implementation, there may be one or more of the followingScaling Policies. (1) Fixed—The number of Stream Segments may not varywith load. (2) Size-based—As the number of bytes of data per secondwritten to the Stream increases past a certain target rate, the numberof Stream Segments may be increased. If it falls below a certain level,the number of Stream Segments may be decreased. (3) Event-based—Similarto Size-based Scaling Policy, except it may use the number of Eventsinstead of the number of bytes.

Events, Stream Segments and Autoscaling

FIG. 10 illustrates an example graph depicting Events and theirrelationship between routing keys, stream segments, and time inaccordance with implementations of this disclosure. In someimplementations, an Event may be written into one of the Stream'sSegments. In some implementations, Stream Segments may be a bucketing ofEvents based on Routing Key and time. In many implementations, Eventspublished to a Stream within a given value of Routing Key may appear inthe same Stream Segment.

In some implementations, Events may be written into the active StreamSegments. In many implementations, Segments that are sealed may notaccept writes. In FIG. 10, at time “now”, Stream Segments 3, 6 and 4 areactive and between those three Stream Segments the entire key space iscovered.

Stream Segments and Reader Groups

FIG. 11 illustrates an example of Stream Segments and associatedReaderGroups in accordance with implementations of this disclosure. Incertain implementations, a Reader in a ReaderGroup may be assigned zeroor more Stream Segments from which to read. In many implementations, thenumber of Stream Segments may be balanced to which Reader it isassigned. In FIG. 11, Reader B1 reads from 2 Stream Segments while eachof the other Readers in the Reader Group have only 1 Stream Segment toread from. In certain implementations, each Stream Segment may be readby one Reader in any ReaderGroup configured to read from that Stream. Inmany implementations, Readers may be added to the ReaderGroup, orReaders crash and may be removed from the ReaderGroup. In manyimplementations Stream Segments so that Stream Segments may bereassigned to be balanced amongst the Readers.

In some implementations, a number of Stream Segments in a Stream maydetermine the upper bound of parallelism of readers within aReaderGroup. In many implementations, the more Stream Segments, the moreseparate, parallel sets of Readers may consume a Stream. In FIG. 11,Stream 1 has 4 Stream Segments. That means that the largest effectiveReader Group would contain 4 Readers. Reader Group named “B” in theabove figure is not quite optimal. If one more Reader was added to theReaderGroup, each Reader would have 1 Stream Segment to process,maximizing read parallelism. However, the number of Readers in theReaderGroup increases beyond 4, at least one of the Readers will not beassigned a Stream Segment. If Stream 1 in the example implementationabove experienced a Scale-Down Event, reducing the number of StreamSegments to 3, then Reader Group B as depicted would have an idealnumber of Readers.

In some implementations, an Auto Scaling feature may enable dynamicallydetermining the right size of a Stream. In most implementations, PravegaStreams may grow and shrink to match the behavior of the data input. Inmost implementations, a size of any Stream may be limited by the totalstorage capacity made available to the Pravega cluster. In manyimplementations, if bigger streams are needed, more storage may be addedto a cluster.

In certain implementations, applications may be made aware of changes ina Stream's Segments. In some implementations, the current disclosure mayprovide hooks to allow applications to react to changes in the number ofSegments in a Stream, adjusting the number of Readers within aReaderGroup, to maintain optimal read parallelism if resources allow. Incertain implementations, this may enable a Flink application to increaseor decrease the number of task instances that are processing a Stream inparallel, as scale events occur over time.

In certain implementations, the current disclosure may enable: (1)Events with the same Routing Key are consumed in the order they werewritten; (2) Events with different Routing Keys sent to a specificReader will always be seen in the same order even if the Reader backs upand re-reads them; and (3) If there are multiple Readers reading aStream and they all backup and replay the Stream, they will each see thesame Events in the same order each time.

ReaderGroup Checkpoints

In certain implementations, the current disclosure may enable ability anapplication to initiate a Checkpoint on a ReaderGroup. In manyimplementations, a Checkpoint may create a consistent “point in time”persistence of the state of each Reader in the ReaderGroup, by using aspecialized Event (a Checkpoint Event) to signal each Reader to preserveits state. In certain implementations, once a Checkpoint has beencompleted, the application may use the Checkpoint to reset all theReaders in the ReaderGroup to the known consistent state represented bythe Checkpoint.

Transactions

FIG. 12 illustrates an example graph depicting Events published into aTransaction and their relationship between routing keys, streamsegments, and time in accordance with implementations of thisdisclosure. In certain implementation, Pravega may support Transactions.In some implementation, a Transaction may “batch” up a bunch of Eventsand commit them as a unit into a Stream. In certain implementations, aFlink job may continuously produce results of some data processing anduse the Transaction to durably accumulate the results of the processing.In certain implementations, at the end of a time window (for example)the Flink job may commit the Transaction and therefore may make theresults of the processing available for downstream processing, or in thecase of an error, abort the Transaction and the results disappear.

In certain implementations, a difference between Pravega's Transactionsand similar approaches (such as Kafka's producer-side batching) isrelated to durability. Events added to a Transaction are durable whenthe Event is ack'd back to the Writer. However, the Events in theTransaction are not visible to readers until the Transaction iscommitted by the Writer. A Transaction is a lot like a Stream; aTransaction is associated with multiple Stream Segments. When an Eventis published into a Transaction, the Event itself is appended to aStream Segment of the Transaction. Say a Stream had 5 Segments, when aTransaction is created on that Stream, conceptually that Transactionalso has 5 Segments. When an Event is published into the Transaction, itis routed to the same numbered Segment as if it were published to theStream itself (if the Event would have been placed in Segment 3 in the“real” Stream, then it will appear in Segment 3 of the Transaction).When the Transaction is committed, each of the Transaction's Segments isautomatically appended to the corresponding Segment in the real Stream.If the Stream is aborted, the Transaction, all its Segments and all theEvents published into the Transaction are removed from Pravega.

In certain implementations, events published into a Transaction may notbe visible to the Reader until that transaction is committed.

State Synchronizers

FIG. 13 illustrates an example state synchronizer in accordance withimplementations of this disclosure. In many implementations, Pravega maybe a streaming storage primitive. In certain implementations, thecurrent disclosure may provide a mechanism to coordinate processes in adistributed computing environment. In certain implementations, a StateSynchronizer feature of Pravega may fall into the latter category.

In certain implementation, a State Synchronizer may use a Pravega Streamto provide a synchronization mechanism for state shared between multipleprocesses running in a cluster, making it easier to build distributedapplications. In most implementations, with a State Synchronizer, thecurrent disclosure may enable reads and changes to be made to sharedstate with consistency and optimistic locking.

In certain implementations, a State Synchronizer may be used to maintaina single, shared copy of an application's configuration property acrossall instances of that application in a cloud. In some implementations, aState Synchronizer may be used to store one piece of data or a map withthousands of different key value pairs. In some implementations, a StateSynchronizer may be used to manage a state of ReaderGroups and Readersdistributed throughout the network.

In some implementation, a State Synchronizer may be created on a Streamin a fashion similar to how an EventWriter is created. In manyimplementations, a State Synchronizer may keep a local copy of theshared state to make access to the data really fast for the application.In certain implementations, changes to shared state may be writtenthrough the State Synchronizer to the Stream keeping track of allchanges to the shared state. In some implementations, each applicationinstance may use a State Synchronizer to stay up to date with changes bypulling updates to shared state and modifying the local copy of thedata. In many implementations, consistency may be maintained through aconditional append style of updates to the shared state through theState Synchronizer, making sure that updates are made only to the mostrecent version of the shared state.

In many implementations, a State Synchronizer may occasionally be“compacted”, compressing and removing old state updates so that only themost recent version of the state is kept in the backing stream. Incertain implementations, a compacting feature may help app developersmake sure that shared state does not grow unchecked.

In some implementations, a State Synchronizer may work best when mostupdates to shared state are small in comparison to the total data sizebeing stored, allowing them to be written as small deltas. In manyimplementations, as with any optimistic concurrency system, a StateSynchronizer may not be at its best when many processes are allattempting to simultaneously update the same piece of data.

Example Component Level Implementation

FIG. 14 illustrates an example implementation of a set of streamingclients, a Pravega Cluster, the autoscaling of segments, and tieredstorage in accordance with implementations of this disclosure.

Pravega clients are Writers and Readers. Writers write Events into aStream. Readers read Events from a Stream. Readers are grouped intoReaderGroups to read from a Stream in parallel.

The Controller is a server-side component that manages the control planeof Pravega. Streams are created, updated and listed using the ControllerAPI.

The Pravega Server or Pravega Node is a server-side component thatimplements read, write and other data plane operations.

Streams are the fundamental storage primitive in Pravega. Streamscontain a set of data elements called Events. Events are appended to the“tail” of the Stream by Writers. Readers can read Events from anywherein the Stream.

A Stream is partitioned into a set of Stream Segments. The number ofStream Segments in a Stream can change over time. Events are writteninto exactly one of the Stream Segments based on Routing Key. For anyReaderGroup reading a Stream, each Stream Segment is assigned to oneReader in that ReaderGroup.

Each Stream Segment is stored in a combination of Tier 1 and Tier 2storage. The tail of the Segment is stored in Tier 1 providing lowlatency reads and writes. The rest of the Segment is stored in Tier 2,providing high throughput read access with near-infinite scale and lowcost.

In many implementations, to deliver an efficient implementation ofStreams, the current disclosure may include a tiered storage model. Inmany implementations, events may be persisted in low latency/high TOPSstorage (Tier 1 Storage) and higher throughput storage (Tier 2 Storage).In most implementations, Writers and Readers may be oblivious to thetiered storage model from an API perspective.

Append Only Log

FIG. 15 illustrates an example append-only log in accordance withimplementations of this disclosure. In some embodiments, write activityand much of the read activity may happen at an end tail of the log. Inmany embodiments, writes may be appended to a log and many this mayenable reads to the data as fast as it arrives in the log. In manyembodiments, data access mechanisms described herein may be dominated bythe need for low-latency—low latency writes by Writers and near realtime access to the published data by Readers.

In certain embodiments, not all Readers may read from the tail of thelog. In some embodiments, certain Readers may want to read starting atsome arbitrary position in a log. In some embodiments, reads from anarbitrary position may be referred to as catch-up reads. Conventionally,access to historical data traditionally was done by batch analyticsjobs, often using HDFS and Map/Reduce. In many embodiments, usingtechniques of the instant disclosure, it may be possible to accesshistorical data as well as current data by just accessing the log. In aparticular embodiment, the historical data may be stored in SSDs. Insome embodiments, the current disclosure may provide a mechanism thatallows customers to use cost-effective, highly-scalable, high-throughputstorage for the historical part of the log, to enable retention ofhistorical data.

In some embodiments, Tier 1 Storage may be used to make writing toStreams fast and durable and to make sure reading from the tail of aStream is as fast as possible. In certain embodiments, Tier 1 Storagemay be implemented on faster SSDs or even non-volatile RAM.

In many embodiments, Tier 2 Storage may provide a highly-scalable,high-throughput cost-effective storage. In some embodiments, Tier 2 maybe deployed on spinning disks. In certain embodiments, the currentdisclosure may enable asynchronous migration of Events from Tier 1 toTier 2 to reflect the different access patterns to Stream data. Incertain embodiments, Tier 2 Storage may be based on an HDFS model.

FIGS. 16-18 illustrate methods and/or flow diagrams in accordance withthis disclosure. For simplicity of explanation, the methods are depictedand described as a series of acts. However, acts in accordance with thisdisclosure can occur in various orders and/or concurrently, and withother acts not presented and described herein. Furthermore, not allillustrated acts may be required to implement the methods in accordancewith the disclosed subject matter. In addition, those skilled in the artwill understand and appreciate that the methods could alternatively berepresented as a series of interrelated states via a state diagram orevents. Additionally, it should be appreciated that the methodsdisclosed in this specification are capable of being stored on anarticle of manufacture to facilitate transporting and transferring suchmethods to computing devices. The term article of manufacture, as usedherein, is intended to encompass a computer program accessible from anycomputer-readable device or storage media.

Moreover, various acts have been described in detail above in connectionwith respective system diagrams. It is to be appreciated that thedetailed description of such acts in the prior figures can be and areintended to be implementable in accordance with one or more of thefollowing methods.

Referring now to FIG. 16, there is illustrated an example flow diagrammethod for de-multiplexing streaming data from a durable log into a setof segment containers and storing data form the segment containers intolong term storage in accordance with implementations of this disclosure.

At 1602, a set of segment containers can be maintained, wherein eachsegment container in the set of segment containers is associated withone or more segments, and each segment is associated with one and onlyone segment container.

At 1604, an operation from the client can be assigned to exactly onesegment container within the set of segment containers using apredefined mapping function. A segment container can multiplexoperations for all the stream segments mapped to that segment containerinto a single durable log, owned by that segment container. The segmentcontainer can also batch a set of operations that arrive at or aroundthe same time to it into a smaller set of larger writes that can beappended to its durable log using a processing queue. In oneimplementation, the segment to segment container mapping function looksat the segment name for each operation and applies a uniformdistribution across all existing segment containers. It can beappreciated that the durable log is able to ingest data for a largernumber of stream segments, with many small writes. It can be appreciatedthat as the name of the stream segment is constant, the hash will alwaysmap to the unique container.

At 1606, a set of operations from a durable log in a segment containercan be read, wherein the durable log contains multiplexed dataassociated with a set of stream segments that are mapped to that durablelog's segment container.

At 1608, the set operations can be de-multiplexed from the durable loginto a set of stream segment buffers based on a stream segmentassociated with each operation in the set of operations in the durablelog. The segment buffer groups streaming data targeted to the uniquestream segment associated with the segment buffer.

At 1610, data can be packaged within a segment among the set ofsegments, within a segment container among the set of segmentcontainers, based on the set of stream segment buffers, in response to apolicy trigger. In one implementation, the policy trigger is based on atleast one of a size of outstanding accumulated data for the segment inthe segment's buffer or a last received operation time associated withthe segment in the segment's buffer. It can be appreciated that as thedurable log is de-multiplexed, the segment buffers related to thestreaming data will fill up with data over time, until a policy triggeroccurs that begins the process of migrating the data from the segmentcontainer into long term stable storage.

At 1612, the packaged data of the segment can be stored in a long termstorage location. In one implementation, the durable log is replicatedand stored in non-volatile storage. In one implementation, the long termstorage location has a higher latency, a higher throughput, and a lowercost per gigabyte than the durable log.

At 1614, the durable log can be truncated based on the packaged data.

The illustrated aspects of the disclosure can be practiced indistributed computing environments where certain tasks are performed byremote processing devices that are linked through a communicationsnetwork. In a distributed computing environment, program modules can belocated in both local and remote memory storage devices.

The systems and processes described above can be embodied withinhardware, such as a single integrated circuit (IC) chip, multiple ICs,an application specific integrated circuit (ASIC), or the like. Further,the order in which some or all of the process blocks appear in eachprocess should not be deemed limiting. Rather, it should be understoodthat some of the process blocks can be executed in a variety of ordersthat are not all of which may be explicitly illustrated herein.

What has been described above includes examples of the implementationsof the present disclosure. It is, of course, not possible to describeevery conceivable combination of components or methods for purposes ofdescribing the claimed subject matter, but many further combinations andpermutations of the subject innovation are possible. Accordingly, theclaimed subject matter is intended to embrace all such alterations,modifications, and variations that fall within the spirit and scope ofthe appended claims. Moreover, the above description of illustratedimplementations of this disclosure, including what is described in theAbstract, is not intended to be exhaustive or to limit the disclosedimplementations to the precise forms disclosed. While specificimplementations and examples are described herein for illustrativepurposes, various modifications are possible that are considered withinthe scope of such implementations and examples, as those skilled in therelevant art can recognize.

In particular and in regard to the various functions performed by theabove described components, devices, circuits, systems and the like, theterms used to describe such components are intended to correspond,unless otherwise indicated, to any component which performs thespecified function of the described component (e.g., a functionalequivalent), even though not structurally equivalent to the disclosedstructure, which performs the function in the herein illustratedexemplary aspects of the claimed subject matter. In this regard, it willalso be recognized that the innovation includes a system as well as acomputer-readable storage medium having computer-executable instructionsfor performing the acts and/or events of the various methods of theclaimed subject matter.

What is claimed is:
 1. A method comprising: maintaining a set of segmentcontainers, wherein each segment container in the set of segmentcontainers is associated with at least one segment, and each segment isassociated with one segment container; assigning operations receivedfrom a client application to segment containers within the set ofsegment containers using a predefined mapping function; reading a set ofoperations from a durable log in a segment container, wherein thedurable log contains multiplexed data associated with a set of streamsegments that are mapped to the segment container; de-multiplexing theset of operations into a set of stream segment buffers based on a streamsegment associated with each operation in the set of operations in thedurable log; packaging data within a segment among the set of segments,within a segment container among the set of segment containers, based onthe set of stream segment buffers, in response to a policy trigger; andstoring the packaged data of the segment in a long-term storagelocation.
 2. The method of claim 1, wherein the durable log isreplicated and stored in non-volatile storage.
 3. The method of claim 2,wherein the long-term storage location has a higher latency, a higherthroughput, and a lower cost per gigabyte than the durable log.
 4. Themethod of claim 1, wherein the policy trigger is based on at least oneof a size of outstanding accumulated date for the segment in thesegment's buffer or a last received operation time associated with thesegment in the segment's buffer.
 5. The method of claim 1, furthercomprising: truncating the durable log based on the packaged data.
 6. Asystem comprising at least one storage device and at least one hardwareprocessor configured to: maintain a set of segment containers, whereineach segment container in the set of segment containers is associatedwith at least one segment, and each segment is associated with onesegment container; assign operations received from a client applicationto segment containers within the set of segment containers using apredefined mapping function; read a set of operations from a durable login a segment container, wherein the durable log contains multiplexeddata associated with a set of stream segments that are mapped to thesegment container; de-multiplex the set of operations into a set ofstream segment buffers based on a stream segment associated with eachoperation in the set of operations in the durable log; package datawithin a segment among the set of segments, within a segment containeramong the set of segment containers, based on the set of stream segmentbuffers, in response to a policy trigger; and store the packaged data ofthe segment in a long-term storage location.
 7. The system of claim 6,wherein the durable log is replicated and stored in non-volatilestorage.
 8. The system of claim 7, wherein the long-term storagelocation has a higher latency, a higher throughput, and a lower cost pergigabyte than the durable log.
 9. The system of claim 6, wherein thepolicy trigger is based on at least one of a size of outstandingaccumulated date for the segment in the segment's buffer or a lastreceived operation time associated with the segment in the segment'sbuffer.
 10. The system of claim 6, further configured to: truncate thedurable log based on the packaged data.
 11. A non-transitory computerreadable medium with program instructions stored thereon, when executedby a processor, cause the processor to perform the following acts:maintaining a set of segment containers, wherein each segment containerin the set of segment containers is associated with at least onesegment, and each segment is associated with one segment container;assigning operations received from a client application to segmentcontainers within the set of segment containers using a predefinedmapping function; reading a set of operations from a durable log in asegment container, wherein the durable log contains multiplexed dataassociated with a set of stream segments that are mapped to the segmentcontainer; de-multiplexing the set of operations into a set of streamsegment buffers based on a stream segment associated with each operationin the set of operations in the durable log; packaging data within asegment among the set of segments, within a segment container among theset of segment containers, based on the set of stream segment buffers,in response to a policy trigger; and storing the packaged data of thesegment in a long-term storage location.
 12. The non-transitory computerreadable medium of claim 11, wherein the durable log is replicated andstored in non-volatile storage.
 13. The non-transitory computer readablemedium of claim 12, wherein the long-term storage location has a higherlatency, a higher throughput, and a lower cost per gigabyte than thedurable log.
 14. The non-transitory computer readable medium of claim11, wherein the policy trigger is based on at least one of a size ofoutstanding accumulated date for the segment in the segment's buffer ora last received operation time associated with the segment in thesegment's buffer.
 15. The non-transitory computer readable medium ofclaim 11, with program instructions stored thereon to further performthe following acts: truncating the durable log based on the packageddata.